System and method for camera control in a surveillance system

ABSTRACT

A method and a system for calibrating a camera in a surveillance system. The method and system use a mathematical rotation between a first coordinate system and a second coordinate system in order to calibrate a camera with a map of an area. In some embodiments, the calibration can be used to control the camera and/or to display a view cone on the map.

The present application is a continuation of U.S. patent applicationSer. No. 12/946,413, filed Nov. 15, 2010, now U.S. Pat. No. 8,193,909,which is a continuation of PCT Application No. PCT/US 10/56691, filedNov. 15, 2010, the full disclosures of which are incorporated byreference herein in their entirety.

TECHNICAL FIELD

The present invention relates to methods of camera control, and moreparticularly to methods of camera control in a surveillance system.

BACKGROUND ART

Surveillance systems use cameras to survey areas of interest. Thecameras produce video feeds that can be displayed to a user. In thismanner, the user can remotely observe an area of interest. In somecases, the cameras in the surveillance system are fixed and the usercannot change their field of view. In other cases, however, the camerasand their field of views can be adjusted through user control.Typically, the control is performed using a manual control, such as ajoystick. Thus, if the video feed includes a moving object, the usermanually controls the camera to follow the moving object. This manualcontrol of the camera is not very efficient, particularly if the userhas a limited time period to capture and/or respond to the movingobject. These problems are further compounded if the camera is notaccurately and properly mounted (e.g., to be level with the ground). Inanother example, image analysis is used to automatically control thecamera to follow an object within the field of view. The shortcomingwith this approach is that the image analysis cannot be performed unlessthe object is within the field of view of the camera. If the movingobject is not within the field of view, the user must manually guide thecamera to capture the moving object within its field of view.

SUMMARY

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for calibrating a surveillance system. Illustrativeembodiments of the present invention include displaying a video feedfrom a camera that has an orientation characterized by pan, zoom, andtilt coordinates. Exemplary embodiments further include displaying a mapof an area that is characterized by geospatial coordinates. A userselects at least three pairs of points using an input device. A firstpoint of the pair is selected in the map and a second point of the pairis selected from the video feed. The first point and the second pointcorrespond to the same geographic location. The three points selected inthe map are converted from geospatial coordinates into Cartesiancoordinates defined by a first coordinate system. The three pointsselected in the video feed are converted from pan, zoom, and tiltcoordinates into Cartesian coordinates defined by a second coordinatesystem. A mathematical rotation between the first coordinate system andthe second coordinate system is then determined based upon the Cartesiancoordinates for the three pairs of points.

Various embodiments of the present invention also include allowing theuser to select at least one point in the map using the input device. Thegeospatial coordinates for the selected point are converted intoCartesian coordinates defined by the first coordinate system and themathematical rotation is applied to those Cartesian coordinates todetermine Cartesian coordinates defined by the second coordinate system.The Cartesian coordinates defined by the second coordinate system arethen further converted into pan and tilt coordinates for the selectedpoint. Orientation instructions are then provided to the camera basedupon the pan and tilt coordinates for the selected point.

In additional or alternative embodiments of the present invention,coordinates from a sensor for a target are received and, if thecoordinates for the target are not Cartesian coordinates defined by thefirst coordinate system, then they are converted into Cartesiancoordinates defined by the first coordinate system. The mathematicalrotation is applied to the Cartesian coordinates defined by the firstCartesian coordinate system to determine Cartesian coordinates definedby the second Cartesian coordinate system. The Cartesian coordinatesdefined by the second Cartesian coordinate system are converted into panand tilt coordinates and orientation instructions are provided to thecamera based upon the pan and tilt coordinates.

In another illustrative embodiment of the present invention, the videofeed has upper left, upper right, lower left, and lower right corners.Exemplary embodiments of the invention further include determiningeffective pan and tilt angles for at least the lower left and lowerright corners of the video feed based upon the pan, zoom, and tiltcoordinates for the camera orientation. The effective pan and tiltangles for at least the lower left and lower right corners of the videofeed are converted into Cartesian coordinates defined by the secondcoordinate system. The mathematical rotation is applied to the Cartesiancoordinates defined by the second coordinate system to determineCartesian coordinates defined by the first coordinate system for atleast the lower left and lower right corners of the video feed. A viewcone is determined using the Cartesian coordinates defined by the firstcoordinate system for at least the lower left and lower right corners ofthe video feed and based upon the upper left and upper right corners ofthe video feed. The view cone is displayed on the map.

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for controlling a camera that has an orientationcharacterized by pan, zoom, and tilt coordinates. Exemplary embodimentsof the invention include displaying a map of an area that ischaracterized by geospatial coordinates. A user selects at least onepoint in the map using the input device. The geospatial coordinates forthe selected point are converted into Cartesian coordinates defined by afirst coordinate system that characterizes the map. A mathematicalrotation is applied to the Cartesian coordinates for the selected pointto determine Cartesian coordinates defined by a second coordinate systemthat characterizes a video feed from the camera. The mathematicalrotation provides a conversion between the first coordinate system andthe second coordinate system. The Cartesian coordinates defined by thesecond coordinate system are converted into pan and tilt coordinates forthe selected point and orientation instructions are provided to thecamera based upon at least the pan and tilt coordinates for the selectedpoint. The video feed from the camera is then displayed according to thepan and tilt coordinates.

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for prioritizing video feeds from a plurality of camerasthat have locations characterized by coordinates. Exemplary embodimentof the invention include receiving coordinates for a point of interestand determining whether the point of interest is within the viewingrange of any of the plurality of the cameras. If more than one camera iswithin viewing range of the point of interest, the distances between thecameras and the point of interest are determined. Various embodiments ofthe present invention further include determining which of the camerasis the least distant from the point of interest. Orientationinstructions are provided to the least distant camera based upon thecoordinates of the point of interest and a video feed is displayed fromthe least distant camera.

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for tracking at least one target using a sensor and acamera that has an orientation characterized by pan, zoom, and tiltcoordinates. Exemplary embodiments of the invention include receivingcoordinates from the sensor for the target. If the coordinates for thetarget are not Cartesian coordinates defined by a first coordinatesystem characterizing the map, then the coordinates are converted intoCartesian coordinates defined by the first coordinate system. Amathematical rotation is applied to the Cartesian coordinates defined bythe first coordinate system in order to determine Cartesian coordinatesdefined by a second coordinate system that characterizes a video feedfrom the camera. The mathematical rotation provides a conversion betweenthe first coordinate system and the second coordinate system. TheCartesian coordinates defined by the second Cartesian coordinate systemare converted into pan and tilt coordinates and orientation instructionsare provided to the camera based upon the pan and tilt coordinates. Thevideo feed from the camera is then displayed according to the pan andtilt coordinates.

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for displaying a view cone for a camera that has anorientation characterized by pan, zoom, and tilt coordinates. Exemplaryembodiments of the present invention include displaying a map of anarea, determining the view cone based upon the pan, zoom, and tiltcoordinates for the camera, and displaying the view cone on the map.

Illustrative embodiments of the present invention are directed to amethod, a system, and a computer readable medium encoded withinstructions for displaying a view cone for a camera that has anorientation characterized by pan, zoom, and tilt coordinates. Exemplaryembodiments of the present invention include displaying a map of an areathat is characterized by geospatial coordinates. Effective pan and tiltangles are determined for at least the lower left and lower rightcorners of the video feed based upon the pan, zoom, and tilt coordinatesfor the camera orientation. The effective pan and tilt angles for atleast the lower left and lower right corners of the video feed areconverted into Cartesian coordinates defined by a second coordinatesystem that characterizes the video feed. A mathematical rotation isapplied to the Cartesian coordinates for at least the lower left andlower right corners of the video feed to determine Cartesian coordinatesdefined by a first coordinate system that characterizes the map. Themathematical rotation provides a conversion between the first coordinatesystem and the second coordinate system. A view cone is then determinedbased upon the Cartesian coordinates, defined by the first coordinatesystem, for at least the lower left and lower right corners and basedupon the upper left and upper right corners of the video feed. The viewcone is then displayed on the map.

When the tilt coordinate for the camera is below the horizon,determining the view cone based upon the upper left and upper rightcorners of the video feed further includes determining effective pan andtilt angles for the upper left and upper right corners of the video feedbased upon the pan, zoom, and tilt coordinates for the cameraorientation. The effective pan and tilt angles for the upper left andupper right corners of the video feed are converted into Cartesiancoordinates defined by the second coordinate system. The mathematicalrotation is applied to the Cartesian coordinates to determine Cartesiancoordinates defined by the first coordinate system for the upper leftand upper right corners of the video feed. The view cone is determinedbased upon the Cartesian coordinates, defined by the first coordinatesystem, for the upper left, upper right, lower left and lower rightcorners.

When the tilt coordinate for the camera is above the horizon,determining the view cone based upon the upper left and upper rightcorners of the video feed includes determining effective tilt angles forthe upper left and upper right corners of the video feed based upon thepan, zoom, and tilt coordinates for the camera. Coordinates, defined bythe first coordinate system, for the upper left and upper right cornersof the video feed are determined based upon a resolvable distance of thecamera. The view cone is determined based upon the Cartesiancoordinates, defined by the first coordinate system, for the upper left,upper right, lower left and lower right corners.

In further exemplary embodiments, the view cone is a polygon and theCartesian coordinates, defined by the first coordinate system, for theupper left, upper right, lower left and lower right corners are thevertices of the polygon.

Illustrative embodiments of the present invention are directed to amethod, a calibrated system, and a computer readable medium encoded withinstructions for controlling a camera. Exemplary embodiments of thepresent invention include displaying a map of an area and determiningCartesian coordinates for a point of interest. The Cartesian coordinatesare defined by a first coordinate system characterizing the map.Exemplary embodiments further include applying a mathematical rotationto the Cartesian coordinates for the point of interest to determineCartesian coordinates defined by a second coordinate system thatcharacterizes a video feed from the camera. The mathematical rotationprovides a conversion between the first coordinate system and the secondcoordinate system. The Cartesian coordinates defined by the secondcoordinate system are converted into at least pan and tilt coordinatesfor the point of interest and orientation instructions are provided tothe camera based upon at least the pan and tilt coordinates for thepoint of interest. The video feed from the camera is displayed accordingto the orientation instructions.

In all or some of the above described embodiments, the geospatialcoordinates are latitude, longitude, and altitude coordinates. The inputdevice is one or more of a mouse, a cursor, a crosshair, a touch screen,and a keyboard. The sensor is one or more of a camera, a radar, and amotion detector.

Furthermore, in all or some of the above described embodiments, the mapof an area is displayed and the map is characterized by geospatialcoordinates. In some embodiments, the location of at least one camera isdisplayed on the map. In additional or alternative embodiments, thelocation of at least one sensor is displayed on the map. Also, in someembodiments, the location of at least one target is displayed on themap.

In all or some of the above described embodiments, the mathematicalrotation is determined by displaying a map of an area that ischaracterized by geospatial coordinates. A user selects at least threepairs of points using an input device. The first point of the pair isselected in the map and a second point of the pair being selected fromthe video feed. The first point and the second point corresponding tothe same geographic location. The three points selected in the map areconverted from geospatial coordinates into Cartesian coordinates definedby the first coordinate system. The at least three points selected inthe video feed are converted from pan, zoom, and tilt coordinates intoCartesian coordinates defined by the second coordinate system. Themathematical rotation between the first coordinate system and the secondcoordinate system is determined based upon the Cartesian coordinates forthe at least three pairs of points. In some embodiments, themathematical rotation is a matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of the invention will be more readily understoodby reference to the following detailed description, taken with referenceto the accompanying drawings, in which:

FIG. 1 shows a surveillance system in accordance with one embodiment ofthe present invention;

FIG. 2 shows a method for calibrating a surveillance system inaccordance with one embodiment of the present invention;

FIG. 3A shows a video feed from a camera in accordance with oneembodiment of the present invention;

FIG. 3B shows a map of a building floor plan in accordance with oneembodiment of the present invention;

FIG. 4A shows a video feed from a camera in accordance with oneembodiment of the present invention;

FIG. 4B shows a video feed from a camera in accordance with oneembodiment of the present invention;

FIG. 5 shows a table of selected points in accordance with oneembodiment of the present invention;

FIG. 6 shows a method for controlling at least one camera in accordancewith one embodiment of the present invention;

FIG. 7 shows a map of an outdoor terrain in accordance with oneembodiment of the present invention;

FIG. 8 shows a method for tracking a target in accordance with oneembodiment of the present invention;

FIG. 9 shows a method for prioritizing video feeds from a plurality ofcameras in accordance with one embodiment of the present invention;

FIG. 10 shows a map of an outdoor terrain with a plurality of cameras inaccordance with one embodiment of the present invention;

FIG. 11 shows a viewing range of a camera in accordance with oneembodiment of the present invention;

FIG. 12 shows a method for displaying a view cone for at least onecamera in accordance with one embodiment of the present invention;

FIG. 13 shows a view cone and a camera icon in accordance with oneembodiment of the present invention.

FIG. 14 shows a view cone and camera icon as applied to a map inaccordance with one embodiment of the present invention;

FIG. 15 shows a video feed and a corresponding view cone in accordancewith one embodiment of the present invention;

FIG. 16 shows a video image transposed into a polygonal view cone inaccordance with one embodiment of the present invention;

FIG. 17 shows how an observable distance is calculated using a righttriangle in accordance with one embodiment of the present invention; and

FIG. 18 shows a view cone in accordance with one embodiment of thepresent invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Illustrative embodiments of the present invention are directed to asurveillance system. FIG. 1 shows a surveillance system 100 inaccordance with one embodiment of the present invention. Illustrativeembodiments of the surveillance system include at least one camera 102.In other embodiments, such as the one shown in FIG. 1, the systemincludes a plurality of cameras 102, 104, 106. The cameras 102, 104, 106are in electronic communication with a processor (e.g., a camera server)via, for example, a communications network 110. The processor 108 isalso in communication with at least one display device such as a monitor112. The display device 112 allows a user to view video feeds from thecameras 102, 104, 106. The system 100 also includes an input device 114such as a computer mouse, a cursor, a crosshair, a keyboard, a menu,and/or a joy stick. In various embodiments, the display device 112 isalso the input 114 device (e.g., touch screen). The input device 114 isin electronic communication with the processor 108 and allows a user tocontrol the cameras 102, 104, 106 and to select points in the videofeeds from each of the cameras.

In illustrative embodiments, the surveillance system also includes ageographic information system 116. The geographic information system 116includes a database of maps of geographic areas (e.g., building floorplans and/or outdoor terrains). Each map includes geospatial coordinatesdefinable by a geographic coordinate system (e.g., latitude, longitude,and altitude). In additional or alternative embodiments, the geospatialcoordinates are definable a Cartesian Coordinate system.

In some embodiments, the processor 108 itself includes the geographicinformation system 116, however, in other embodiments, the geographicinformation system may be in communication with the processor via thecommunications network (e.g., internet) 110. The display device 112 isused to display the map and the input device 114 is used to select apoints on the map. In various exemplary embodiments, a first displaydevice may be used to display video feeds from the cameras 102, 104, 106and a second display device is used to display the map. Additionally oralternatively, the surveillance system 100 may include at least onesensor 118 such as a camera, a radar, and/or a motion detector. Thesensor 118 is in electronic communication with the processor 108 via,for example, the computer network 110.

Calibration

Illustrative embodiments of the present invention are directed to amethod for calibrating a surveillance system (such as the one shown inFIG. 1.). FIG. 2 shows a method 200 for calibrating a surveillancesystem in accordance with one embodiment of the present invention. Themethod includes displaying a video feed from a camera using a displaydevice 202. FIG. 3A shows a video feed 301 from a camera in accordancewith one embodiment of the present invention. In the present example,the video feed 301 shows the breeze way of a building. In illustrativeembodiments, the camera is a PTZ camera with an orientation thatcharacterized by pan, zoom, and tilt coordinates, however, the use ofother cameras is also within the scope of the present invention.

The method also includes using the display device to display a map of anarea 204. FIG. 3B shows a map 300 of a building floor plan in accordancewith one embodiment of the present invention. The floor plan includes ahallway that is located between two rooms. The map 300 is characterizedby geospatial coordinates. In other words, each point on the map 300 canbe defined in terms of a geospatial coordinates such as latitude andlongitude. In some embodiments of the present invention, the map 300does not include a coordinate for altitude. In such an embodiment, thealtitude is assumed to be “0”. In other illustrative embodiment of thepresent invention, every point on the map 300 is defined by an altitudecoordinate and, therefore, every point on the map is defined by threecoordinates: latitude, longitude, and altitude. In yet anotherembodiment of the invention, only some of the points on the map 300 aredefined in terms of altitude. In such an embodiment, the altitude forpoints that are not defined can be extrapolated from adjacent pointsthat do have defined altitudes. For example, if a first point at 30.000degrees latitude and 20.000 degrees longitude does not have a definedaltitude, then points closest to the first point can be used toextrapolate the altitude for the first point. In such an example, themethod uses a second point, at 30.000 degrees latitude and 19.999degrees longitude with an altitude of 4 feet, and a third point, at30.000 degrees latitude and 20.001 degrees longitude with an altitude of10 feet, to assume that the altitude for the first point is about 7feet. In illustrative embodiments, various other extrapolationtechniques can also be used. For example, in some embodiments,algorithms for smoothing the extrapolated altitudes may be used.

Illustrative embodiments of the map 300 also show the location of acamera on the map using a camera icon 302. In this manner, a user canview the map 300 and understand the vantage point and perspective of thevideo feed from the camera. In FIG. 3B, a camera icon 302 shows thelocation of the camera at the end of the hallway.

The map 300 also displays an input device 304 so that the user canselect points on the map 300. In FIG. 3B, the input device 304 is acrosshair, however, in other embodiments the input device can take theform of, for example, an arrow, a touch screen, and/or a menu.

Illustrative embodiments of the calibration method allow the user toselect at least three pairs of points using the input device 304, 206. Afirst point of the pair is selected in the map 300 using the inputdevice 304. In FIG. 3B, the user selects a left corner 306 in thehallway using the crosshair 304. The processor registers the particulargeospatial coordinates for the selected point. Once the user selects thefirst point in the map 300, he then selects the second point of the pairfrom a video feed. FIG. 4A shows a video feed 400 in accordance with oneembodiment of the present invention. The video feed 400 and the map 300may be simultaneously displayed in separate windows on one displaydevice or each separately on its own display device. In FIG. 4A, theuser uses a crosshair 402 to select the left corner 306 in the hallwayin the video feed 400. When the user selects a point in the video feed400, a processor queries the camera for its orientation (e.g., panangle, tilt angle, and zoom). Once the orientation is received from thecamera, the processor registers the particular orientation of thecamera. The first and second points in the pair should be selected sothat they correspond to the same geographic location. In this manner,the method appropriately calibrates the video feed 400 to the map 300.

Once the first pair of points is selected, the user selects a secondpair of points. In FIG. 3B, the user selects a first point of the pairas a right corner 306 in the hallway. The second point of the pair isthen selected in the video feed 400. FIG. 4B shows the video feed 400 inaccordance with one embodiment of the present invention. In FIG. 4B, theuser has shifted the orientation of the camera and, therefore, the videofeed 400 displays a different part of the hallway. The user uses thecrosshair 402 to select the right corner 308 of the hallway as itappears in the video feed 400.

Once the second pair of points is selected, the user selects a thirdpair of points. In illustrative embodiments of the present invention,the user selects three pairs of points, however, in other exemplaryembodiments, the user may selects 4, 5, or more pairs of points for thecalibration. In some exemplary embodiments, using more pairs of pointsresults in a more accurate calibration because any error within thecoordinates of one point is averaged over a greater number of points.

In illustrative embodiments of the present invention, as describedabove, the user selects the first point of the pair in the map 300 andthen the second point of the pair in the video feed 400. Once the firstpair of points is selected, the user then selects the next pair ofpoints. The embodiments of the present invention, however, are notlimited to this particular order. In fact, the points can be selected inany order so long as there are at least three pairs of points selected.For example, in one illustrative embodiment, the user first selects apoint of the pair in the video feed 400 and then selects the other pointof the pair in the map 300. In yet another illustrative embodiment, theuser first selects all three points in the video feed 400 and thenselects corresponding points in the map 300.

FIG. 5 shows a table 500 of selected points in accordance with oneembodiment of the present invention. The table includes 6 pairs ofpoints and lists the particular coordinates for each of the pairs.Columns 502 and 504 list the geospatial coordinates for the pointsselected in the map 301. Columns 506 and 508 list the cameraorientations for the points selected in the video feed 400. Thegeospatial coordinates in the table 500 are latitude, longitude, andaltitude (altitude not displayed), while the coordinates for the cameracoordinates are pan angle, tilt angle, and zoom (zoom not displayed).

In various embodiments of the calibration method, the pan, tilt, andzoom orientation of the camera for each point in the video feed areconverted into effective pan and tilt angles for the points themselves.The processor initially registers the orientation of the camera itself,but these initial pan and tilt angles are representative of the centerof the video feed. When a user selects points away from the center ofthe video feed 400, the effective pan and tilt angles for the pointswill be different from the pan and tilt orientation of the cameraitself. To determine the effective pan and tilt angles for the points, ahorizontal angular field of view (H_(FOV)) and vertical angular field ofview (V_(FOV)) is determined using Equations 1 and 2:

$\begin{matrix}{H_{FOV} = {Zoom}} & (1) \\{V_{FOV} = {{Zoom} \times \frac{1}{AspectRatio}}} & (2)\end{matrix}$

The effective pan angle (θ) can be calculated using Equation 3,

$\begin{matrix}{\theta = {{H_{FOV} \times \frac{H_{D}}{W_{Videofeed}}} + {Pan}}} & (3)\end{matrix}$

wherein H_(D) is the horizontal distance (e.g., in pixels) from thecenter of the video feed 400 to the point, and W_(Videofeed) is thewidth of the video feed (e.g., in pixels). Also, the effective tiltangle (φ) can be calculated using Equation 4,

$\begin{matrix}{\varphi = {{V_{FOV} \times \frac{V_{D}}{H_{Videofeed}}} + {Tilt}}} & (4)\end{matrix}$wherein V_(D) is the vertical distance (e.g., in pixels) from the centerof the video feed 400 to the point, and H_(Videofeed) is the height ofthe video feed (e.g., in pixels).

Illustrative embodiments of the calibration method also includeconverting the at least three points selected in the video feed 400 fromeffective pan and tilt angles to Cartesian coordinates defined by aCartesian coordinate system 210. In various embodiments of thecalibration method, the effective pan and tilt angles are converted tounit vectors on a “camera” Cartesian coordinate system that has itsorigin at the camera's location. The effective pan and tilt angles canbe converted into Cartesian coordinates using Equations 5, 6, and 7:u=cos(φ)×sin(θ)  (5)v=cos(φ)×cos(θ)  (6)w=sin(φ)  (7)

In this manner, Cartesian coordinates are calculated for the effectivepan (θ) and tilt (φ) angles of each point selected in the video feed400.

Illustrative embodiments of the calibration method further includeconverting the three points selected in the map 300 from geospatialcoordinates into Cartesian coordinates defined by a coordinate system208. The coordinate system can be created using a geospatial locationfor a point (P_(lat), P_(lon), P_(alt)) and a geospatial location forthe camera (C_(lat), C_(lon), C_(alt)). The origin of the “map”Cartesian coordinate system is set as the location of the camera. Inillustrative embodiments, the x-axis of the coordinate system pointseast, the y-axis points north, and the z-axis points in a direction awayfrom and normal to the Earth.

The distance from the camera to the point along the x-axis can becalculated using Equation 8:X _(P) =R _(E)×sin(P _(lat) −C _(lat))+(R _(E)×cos(P _(lat)))×(1−cos(P_(lon) ×C _(lon)))×sin(C _(lat))  (8)

The distance from the camera to the point along the y-axis can becalculated using Equation 9:Y _(P) =R _(E)×cos(P _(lat)))×sin(P _(lon) −C _(lon)))  (9)

The distance from the camera to the point along the z-axis can becalculated using Equation 10:Z _(P) =−C _(alt) =−P _(alt)  (10)

In each of Equations 8, 9, and 10, R_(E) is the semi-major axis of theearth, which can be calculated according to Equation 11,R _(E)=(1−K _(flat))×sin²(P _(lat))×R _(a)  (11)

wherein K_(flat) is the Earth flatness constant, which is approximately1/298.257, and R_(a) is the radius of the Earth, which is approximately6,378,137 meters. Accordingly, using Equations 8, 9, and 10, theCartesian coordinates for the point are (X_(P), Y_(P), Z_(P)) and theCartesian coordinates for the camera are (0, 0, 0).

In illustrative embodiments, the three points selected from the videofeed are then converted into unit vectors. The unit vectors can bedetermined using Equations 12, 13, and 14:u=X _(P)/√{square root over (X _(P) ² Y _(P) ² Z _(P) ²)}  (12)v=Y _(P)/√{square root over (X _(P) ² Y _(P) ² Z _(P) ²)}  (13)w=Z _(P)/√{square root over (X _(P) ² Y _(P) ² Z _(P) ²)}  (14)

Accordingly, each of the three points selected from the map is definedas a unit vector (u, v, w) of the “map” Cartesian coordinate system.

In the illustrative embodiments discussed above, the map 300 ischaracterized by geospatial coordinates such as latitude, longitude, andaltitude. In other embodiments, however, the map is alreadycharacterized by a Cartesian coordinate system (before the abovedescribed calibration process) and, therefore, a conversion fromgeospatial coordinates to a Cartesian coordinate system may not benecessary.

Illustrative embodiments of the present invention further includedetermining a mathematical rotation between the “map” coordinate systemand the “camera” coordinate system based upon the Cartesian coordinatesfor the at least three pairs of points selected from the map and thevideo feed 212. The mathematical rotation provides a conversion frompoints in the “map” coordinate system to corresponding points in the“camera” coordinate system. The transform of the rotation provides aconversion from points in the “camera” coordinate system tocorresponding points in the “map” coordinate system. Or vice versa, atransform of the mathematical rotation provides a conversion from pointsin the “camera” coordinate system to corresponding points in the “map”coordinate system, and its transform provides a conversion from pointsin the “map” coordinate system to corresponding points in the “camera”coordinate system. As used in the present application, the term“rotation” also includes a transform of the rotation.

It is known in the art how to develop a mathematical rotation (and itstransform) between a first Cartesian coordinate system and a secondCartesian Coordinate system. For example, in various embodiments, themathematical rotation is a four-by-four matrix that is applied to unitvectors that define a point in one of the “camera” and “map” coordinatesystems. The four-by-four rotation is determined by developing aninitial four-by-four matrix. The initial four-by-four matrix isdeveloped according to, for example, Section 4 of Berthold K. P. Horn,“Closed-form Solution of Absolute Orientation Using Unit Quaternions” 4Journal of the Optical Society of America 629 (April 1987). The Hornarticle is hereby incorporated by reference in its entirety.Eigenvectors for the initial four-by-four matrix are then calculatedusing a Jacobi Eigenvalue algorithm as known in the art. The calculationof the Eigenvalues can be software coded as disclosed in, for example,Section 11.1 of William H. Press et al., “Numerical Recipes in C”(Cambridge University Press 2d ed.) (1992). Section 11.1 of that book isalso hereby incorporated by reference. Once the eigenvectors arecalculated, an eigenvector corresponding to the most positive eigenvalueis selected as the rotation matrix. In illustrative embodiments, therotation matrix is stored by the processor as a quaternion.

Illustrative embodiments of the present invention are advantageous overmany prior art systems because there is no need for the camera to beaccurately aligned with other reference points. For example, many priorart systems require pan-zoom-tilt cameras to be level with the groundwhen installed and/or to be properly aligned with a particularcoordinate system. If not properly aligned, the camera will notaccurately shift and capture points of interest. The inventors haveovercome this problem by calibrating the camera using a mathematicalrotation between a first Cartesian coordinate system and a secondCartesian coordinate system. In illustrative embodiments of the presentinvention, the inventors discovered that the above disclosed calibrationaccounts for any misalignment of the camera with the ground orcoordinate system. In the past, calibration of a camera with a map of anarea was not approached as a problem that could be solved through theuse of a mathematical rotation between two Cartesian coordinate systems,perhaps, because latitude-longitude coordinates and the motion of apan-zoom-tilt camera does not correspond with a Cartesian coordinatesystem.

Camera Control Using the Map

Illustrative embodiments of the present invention are also directed to acomputer-implemented method for controlling a camera in a surveillancesystem (such as the system shown in FIG. 1). FIG. 6 shows a method 600for controlling at least one camera in accordance with one embodiment ofthe present invention. The method includes displaying a map of an areausing a display device 602. FIG. 7 shows a map 700 of an outdoor terrainin accordance with one embodiment of the present invention. The map 700is characterized by geospatial coordinates such as latitude, longitude,and altitude. The method further includes allowing a user to select apoint of interest in the map 700 using an input device 604. In theembodiment of FIG. 6, the input device is a crosshair 702. Inillustrative embodiments of the present invention, the orientation of acamera 704 then shifts so that the camera observes the point ofinterest. In this manner, the user controls the camera 704 and itsorientation by simply selecting a point in the map 700 using thecrosshair 702. Thus, the user can use a geospatial interface to automatecamera control. In the example of FIG. 7, the user is interested inactivity on a road 706 that runs through the map 700. By selecting apoint on the road 706 in the map 700, the camera 704 automaticallyshifts its orientation to the road and the user can view the activity onthe road via the video feed from the camera.

In illustrative embodiments, once the user selects the point on the map,a processor converts the geospatial coordinates for the selected pointinto Cartesian coordinates defined by a coordinate system 606. Inexemplary embodiments, the conversion is performed using Equations 8, 9,10, 11, 12, 13, and 14, as described in the “Calibration” section ofpresent application. Accordingly, the geospatial coordinates for thepoint are defined as unit vectors of a “map” Cartesian coordinatesystem. The processor 108 then applies a mathematical rotation to the“map” Cartesian coordinates for the selected point to determineCartesian coordinates defined by a “camera” Cartesian coordinate system608. The “camera” coordinate system characterizes a video feed from thecamera and a mathematical rotation (or its transform) provides aconversion between the “map” coordinate system and the “camera”coordinate system. In illustrative embodiments, the mathematicalrotation is developed according to the method described in the“Calibration” section of the present application.

The Cartesian coordinates for the selected point (e.g., unit vectors u,v, w) are then defined in terms of pan, zoom, and tilt coordinates(e.g., pan (θ) and tilt (φ)) 610. The Cartesian coordinates (defined bythe “camera” coordinate system) can be converted to pan and tilt anglesusing Equations 15 and 16:

$\begin{matrix}{\theta = {{atan}^{2}\left( {u,v} \right)}} & (15) \\{\varphi = {{atan}\left( \frac{w}{\sqrt{u^{2} + v^{2}}} \right)}} & (16)\end{matrix}$

Accordingly, the selected point is defined in terms of a pan and tiltangles of the camera. Based on these pan and tilt angles, the processorcan provide orientation instructions to the camera so that the camerashifts to view the selected point and the video feed of the selectedpoint is thus displayed to the user 612, 614.

Target Tracking

Illustrative embodiments of the present invention are also directed to acomputer-implemented method for tracking a target using at least onecamera and at least one sensor (e.g., such as those shown in FIG. 1).FIG. 8 shows a method 800 for tracking a target in accordance with oneembodiment of the present invention. The method includes receivingcoordinates for the target at a processor from the sensor 802. In someembodiments, the coordinates for the target that are received from thesensor and the coordinates are already defined in terms of a Cartesiancoordinate system that characterizes a map of an area. For example, insome embodiments, the Cartesian coordinates for a motion detector in themap are determined when the detector is placed in the area. Therefore,if the motion detector is set-off, then the processor can assume thatthe target is located near the coordinates of the motion detector anduses the coordinates of the sensor defined by the “map” coordinatesystem to determine an appropriate camera action. In other embodiments,however, the coordinates for the target that are received from thesensor are defined in terms of a geospatial coordinate system. In suchan embodiment, the geospatial coordinates are converted into Cartesiancoordinates defined by the “map” coordinate system 804. In exemplaryembodiments, the conversion is performed using Equations 8, 9, 10, 11,12, 13, and 14, as described in the “Calibration” section of presentapplication. In this manner, the geospatial coordinates for the targetare defined as unit vectors (u, v, w) of the “map” Cartesian coordinatesystem.

The method for target tracking also includes applying a mathematicalrotation (or its transform) to the “map” Cartesian coordinates for thetarget to determine Cartesian coordinates defined by a “camera”Cartesian coordinate system 806. The “camera” coordinate system definesa video feed from the camera and the mathematical rotation provides aconversion between the “map” coordinate system and the “camera”coordinate system. In illustrative embodiments, the mathematicalrotation is developed according to the method described in the“Calibration” section of the present application.

The Cartesian coordinates for the target (u, v, w) are then defined interms of pan, zoom, and tilt coordinates (e.g., pan “θ” and tilt “φ”)for the camera 808. In various illustrative embodiments, the Cartesiancoordinates are converted to pan and tilt angles using Equations 15 and16, as described in the “Camera Control Using the Map” section ofpresent application. Accordingly, the target is defined in terms of apan and tilt angles of the camera. Based on these pan and tilt angles,the processor provides orientation instructions to the camera so thatthe camera shifts to view the target 810, 812.

In illustrative embodiments, the processor continuously receivescoordinates from the sensor and iteratively determines camerainstructions based upon the new coordinates. In this manner, theprocessor and the camera automatically track the target as it movesthrough the area.

Illustrative embodiments of the target tracking method may also displaya map that shows the location of the sensor using a sensor icon.Furthermore, in additional or alternative embodiments, the map displaysthe location of the target in the area using a target icon. Theprocessor can use the geospatial and/or “camera” Cartesian coordinatesfor the sensor and/or target to display them on the map. By displayingthe target, sensor, and cameras on the map, the user can betterunderstand the location of the target in relation to the sensors andcameras.

Prioritization of Cameras

Illustrative embodiments of the present invention are also directed to acomputer-implemented method for prioritizing video feeds from aplurality of cameras (such as the ones shown in FIG. 1). FIG. 9 shows amethod 900 for prioritizing video feeds from a plurality of cameras inaccordance with one embodiment of the present invention, while FIG. 10shows a map 1000 of an outdoor terrain including a plurality of cameras1002, 1004, 1006, 1008 in accordance with another embodiment of thepresent invention.

Illustrative embodiments of the method include receiving geospatialcoordinates for a point of interest 1010, 902. In the embodiment shownin FIG. 10, the point of interest 1002 is selected by a user from themap 1000 using a display device and an input device. In otherillustrative embodiments, the coordinates for the point of interest arerepresentative of a target and the coordinates are received from asensor. Once the coordinates for the point of interest are received, invarious embodiments, one or more cameras 1002, 1004, 1006, 1008 shifttheir orientation to capture the point of interest 1010. In illustrativeembodiments of the present invention, the cameras 1002, 1004, 1006, 1008are assigned a prioritization based upon their distance from the pointof interest 1010.

Various exemplary embodiments of the prioritization method includedetermining whether the point of interest 1010 is within the viewingrange of any of the cameras 1002, 1004, 1006, 1008, 904. In someexemplary embodiments, the method also includes determining whichcameras 1002, 1004, 1006, 1008 are available for viewing the point ofinterest (e.g., some cameras may be tracking a different target or theuser may not have permission to use certain cameras).

FIG. 11 shows a viewing range 1100 of a camera 1102 in accordance withone embodiment of the present invention. As shown in FIG. 11, theviewing range 1100 of the camera 1102 can also be defined to account forobstacles 1104, 1106, 1108. For example, in FIG. 11, the viewing range1100 of the camera 1102 is blocked by a wall 1108 and, accordingly, thearea 1110 that appears behind the wall is omitted from the viewingrange. Similarly, a house 1106 is also in the way of the viewing range1100 and thus the area 1112 behind the house is also omitted from theviewing range.

In various exemplary embodiments, the viewing range for each camera isdefined by the user. To this end, in illustrative embodiments, the usercan use an input device to outline the viewing range 1100 for the camera1102 within a displayed map, as shown in FIG. 11. In illustrativeembodiments, the user can outline the viewing range of the camera bygenerating a plurality of line segments having starting points and endpoints. The user then closes the line segments to form a polygon thathas vertices defined by the starting points and endpoints of the linesegments. In this manner, the viewing range of the camera can be definedas relatively complex and irregular area, such as the viewing range 1100shown in FIG. 11. In some cases, it may be appropriate to represent theviewing range of the camera using a plurality of polygons. In variousexemplary embodiments, the vertices of the polygon are defined in termsof geospatial coordinates, however, in other embodiments, the verticesare defined in terms of a “map” Cartesian coordinate system or even a“camera” coordinate system.

In exemplary embodiments, to determine whether the point of interest1010 is within the viewing range of any of the cameras 1002, 1004, 1006,1008, the method determines whether the point of interest appears withinthe polygonal viewing range of each of the cameras. To this end,illustrative embodiments of the present invention use variouspoint-in-polygon algorithms known in the art. In one example of such analgorithm, the method assumes a horizontal line starting at the point ofinterest and continuing across the map. If the assumed horizontal lineintersects the line segments of the polygon an even number of times(e.g., 0, 2, 4), then the point of interest is outside the polygon. Ifthe assumed line intersects the polygon an odd number of times (e.g., 1,3, 5), then the point of interest is within the polygon. In anotherexample, the sum of the angles made between the point of interest andeach vertex (or point) making up the polygon are computed. If the sum is360 degrees (e.g., 2π), then the point of interest is interior to thepolygon. If the sum is 0 degrees, then the point of interest is exteriorto the polygon and, therefore, also exterior to the viewing range of thecamera. Further details of point-in-polygon algorithms can be found inthe reference: Paul Bourke, “Determining If a Point Lies on the Interiorof a Polygon” (November 1987) (accessible athttp://local.wasp.uwa.edu.au/˜pbourke/geometry/insidepoly/), which ishereby incorporated by reference in its entirety.

In one exemplary case, if only one camera has the viewing range tocapture the point of interest, the processor provides orientationinstructions to that camera to capture the point of interest. In anotherexample, if there are at least two cameras with the viewing range tocapture the point of interest, the processor determines the distancesbetween the two cameras and the point of interest 906. If the cameralocations and the point of interest are both defined in terms of aCartesian coordinate system (e.g., the “map” Cartesian coordinate systeman/or the “camera” Cartesian coordinate system discussed above), thenthe distance between the cameras and the point of interest can becalculated according to Equation 17,D=√{square root over ((X _(C) −X _(P))²+(Y _(C) −Y _(P))²+(Z _(C) −Z_(P))²)}{square root over ((X _(C) −X _(P))²+(Y _(C) −Y _(P))²+(Z _(C)−Z _(P))²)}{square root over ((X _(C) −X _(P))²+(Y _(C) −Y _(P))²+(Z_(C) −Z _(P))²)}  (17)

wherein (X_(C), Y_(C), Z_(C)) are the Cartesian coordinates for thecamera and (X_(P), Y_(P), Z_(P)) are the Cartesian coordinates for thepoint of interest. Equation 17 is particularly accurate for use withsurveillance systems that do not need to account for the curvature ofthe earth (e.g., surveillance systems within buildings). If the cameralocations and the point of interest are both defined in terms of ageospatial coordinate system (e.g., latitude, longitude, and altitude),then the Cartesian distance can be calculated using Equations 8, 9, 10,and 11 as described in the “Calibration” section of the presentapplication. The distance can then be calculated using Equation 17 asdescribed in this section. The benefit of using Equations 8, 9, 10, and11 is that they account for the curvature of the Earth and, thus,provide better accuracy for surveillance systems that survey, forexample, outdoor terrains. Further details of calculating distancesbetween geospatial coordinates can be found in the reference: Ivan S.Ashcroft, “Projecting an Arbitrary Latitude and Longitude onto a TangentPlane” Brigham Young University (Jan. 21, 1999), which is herebyincorporated by reference in its entirety.

Once the distances between the point of interest and the cameras aredetermined, the processor determines which of the cameras is the leastdistance from the point of interest 908. In illustrative embodiments ofthe present invention, the least distance camera is then provided withorientation instruction based upon the coordinates of the point ofinterest and the video feed from the least distance camera is displayedon a display device 910.

Illustrative embodiments of the present invention are particularlybeneficial when a surveillance system includes a plurality of camerasbut only has a limited bandwidth and/or a limited number of displaydevices to display video feeds. In this case, exemplary embodiments ofthe present invention allow the system to prioritize the cameras anddisplay the video feed from the camera that is closest to the point ofinterest and has the best view of the point of interest.

Furthermore, illustrative embodiments of the present invention no longerrely on the user's manual control to capture a point of interest or amoving object. For example, if a target is walking in a field andsuddenly hides behind a wall, then the target might not be within theviewing range of a camera located in front of the wall, but the targetmight still be within the viewing range of another camera located behindthe wall. In some prior art systems, the user would need to manuallytoggle between the two cameras in order to capture the target behind thewall. Illustrative embodiments of the present invention, however,automatically prioritize and display video feeds based on the camera'srange of view and distance from the target.

View Cone

Illustrative embodiments of the present invention are also directed to acomputer-implemented method for displaying a view cone for a camera.FIG. 12 shows a method 1200 for displaying a view cone for at least onecamera in accordance with one embodiment of the present invention. FIG.13 shows a view cone 1300 and a camera icon 1302 in accordance with oneembodiment of the present invention. In exemplary embodiments of thepresent invention, the view cone 1300 is representative of the field ofview of the camera. Using the view cone 1300 is particularlyadvantageous when it is applied to a map because it helps the userunderstand the vantage point and perspective of the camera within anarea. FIG. 14 shows a view cone 1400 and camera icon 1402 as applied toa map 1404 in accordance with one embodiment of the present invention1202.

In illustrative embodiments of the present invention, the view cone iscreated by transposing the effective coordinates of the upper left,upper right, lower left, and lower right corners of a video feed onto amap. FIG. 15 shows a video feed 1500 and field of view cone 1502 inaccordance with another embodiment of the present invention. The videofeed 1500 includes upper left 1504, upper right 1506, lower left 1508,and lower right corners 1510. The view cone 1502 also includes upperleft 1512, upper right 1514, lower left 1516, and lower right vertices1518. In illustrative embodiments of the present invention, the viewcone 1502 is created by transposing the effective coordinates of theupper left 1504, upper right 1506, lower left 1508, and lower rightcorners 1510 of the video feed 1500 onto upper left 1512, upper right1514, lower left 1516, and lower right vertices 1518 of the view cone1502. The advantage of creating the view cone based upon the effectivecoordinates of the video feed is that, when the orientation of thecamera shifts, the view cone 1502 on the map also changes to betterrepresent the field of view of the camera.

In illustrative embodiments of the present invention, the effectivecoordinates of the upper left 1504, upper right 1506, lower left 1508,and lower right corners 1510 of the video feed 1500 are determined fromthe orientation of the camera (e.g., the pan, zoom, and tilt coordinatesfor the camera). For each of the upper left 1504, upper right 1506,lower left 1508, and lower right corners 1510 of the video feed 1500,effective pan and tilt angles are calculated 1204. To determine theeffective pan and tilt angles for the points, a horizontal angular fieldof view (H_(FOV)) 1520 and vertical angular field of view (V_(FOV)) 1522is determined using, for example, Equations 1 and 2, as described in the“Calibration” section of the present application. Next, the effectivepan and tilt angles (θ, φ) for each of the corners 1504, 1506, 1508, and1510 is determined using, for example, Equations 3 and 4 as describedabove in the “Calibration” section of the application.

In exemplary embodiments of the present invention, the effective pan andtilt angles (θ, φ) for the corners 1504, 1506, 1508, and 1510 areconverted to Cartesian coordinates defined by a Cartesian coordinatesystem 1206. In various embodiments of the method, the effective pan andtilt angles are converted to unit vectors (u, v, w) on a “camera”Cartesian coordinate system that has its origin at the camera'slocation. The effective pan and tilt angles can be converted intoCartesian coordinates using Equations 5, 6, and 7, as described in the“Calibration” section of the present application.

To transpose the corners 1504, 1506, 1508, and 1510 of the video feed1502 to corners 1512, 1514, 1516, and 1512 of the view cone 1500 on themap, a mathematical rotation (or its transform) is applied to the“camera” Cartesian coordinates for the corners 1504, 1506, 1508, and1510, 1208. The mathematical rotation (or its transform) determines unitvectors (u, v, w) defined by a “map” Cartesian coordinate system. The“map” coordinate system defines the map and the mathematical rotationprovides a conversion between the “camera” coordinate system and the“map” coordinate system. In illustrative embodiments, the mathematicalrotation is developed according to the method described in the“Calibration” section of the present application. After the mathematicalrotation, the unit vectors (u, v, w) are converted to scaled Cartesiancoordinates using Equations 18, 19, and 20:X _(P) =u×ScalingFactor  (18)Y _(P) =v×ScalingFactor  (19)Z _(P) =w×ScalingFactor  (20)

In various embodiments of the present invention, the scaling factor canbe determined using Equation 21,

$\begin{matrix}{{ScalingFactor} = {\frac{C_{alt}}{w}}} & (21)\end{matrix}$

wherein, in various embodiments, C_(alt) is the altitude of the cameraabove the ground. In illustrative embodiments of the present invention,the scaled “map” Cartesian coordinates (X_(P), Y_(P), Z_(P)) for thecorners 1504, 1506, 1508, and 1510 are used to determine the view cone1500 and the view cone is then displayed on the map 1210, 1212. In morespecific illustrative embodiments, as shown in FIG. 15, the view cone isa polygon and the “map” Cartesian coordinates for the corners 1504,1506, 1508, and 1510 are used to define the vertices 1512, 1514, 1516,and 1512 of the view cone 1500.

In yet another embodiment of the present invention, the scaled “map”Cartesian coordinates (X_(P), Y_(P), Z_(P)) for the corners 1504, 1506,1508, and 1510 are converted into geospatial coordinates (P_(lat),P_(lon), P_(alt)). In various exemplary embodiments, the “map” Cartesiancoordinates (X_(P), Y_(P), Z_(P)) can be converted into geospatialcoordinates (P_(lat), P_(lon), P_(alt)) by determining deltas forlongitude and latitude using Equations 22 and 23,

$\begin{matrix}{\Delta_{lat} = {\arcsin\frac{X_{P}}{R_{lon}}}} & (22) \\{\Delta_{lon} = {\arcsin\frac{Y_{P} - \left( {1 - {{\cos\left( \Delta_{lat} \right)} \times {\sin\left( C_{lat} \right)} \times R_{lon}}} \right.}{R_{E}}}} & (23)\end{matrix}$

wherein C_(lat) is the known latitudinal geospatial coordinate for thecamera, R_(E) can be calculated from Equation 11 above, and R_(lon) isunknown, but can be estimated as approximately equal to the longitudinalcoordinate of the camera “C_(lon)”. Further details for convertingCartesian coordinates into geospatial coordinates can be found in thereference: Ivan S. Ashcroft, “Projecting an Arbitrary Latitude andLongitude onto a Tangent Plane” Brigham Young University (Jan. 21,1999).

In illustrative embodiments of the present invention, once the deltas“Δ_(lat) and Δ_(lat)” are determined, they can be used to determine thegeospatial coordinates (P_(lat), P_(lon), P_(alt)) for the corners 1504,1506, 1508, and 1510 using Equations 24, 25, and 26,P _(lat) =C _(lat)+Δ_(lat)  (24)P _(lon) =C _(lon)+Δ_(lon)  (25)P _(alt) =C _(alt)  (26)

wherein (C_(lat), C_(lon), C_(alt)) are the known geospatial coordinatesof the camera. Accordingly, in such an embodiment, the geospatialcoordinates (P_(lat), P_(lon), P_(alt)) for the corners 1504, 1506,1508, and 1510 are used to determine the view cone 1500 and the viewcone is then displayed on the map. In more specific illustrativeembodiments, the view cone is a polygon and the geospatial coordinatesfor the corners 1504, 1506, 1508, and 1510 are used to define thevertices 1512, 1514, 1516, and 1512 of the view cone 1500. FIG. 16 showsa video image 1600 with four corners transposed into a polygonal viewcone 1602 with four vertices in accordance with one embodiment of thepresent invention.

In further illustrative embodiments of the present invention, the upperleft 1512 and upper right vertices 1514 of the view cone 1500 aredetermined in a different manner. In some cases, when the tilt angle forthe camera (or the effective tilt angle for at least one of the cornersof the video feed) is above the horizon (e.g., φ is greater than 0), thetransposition of the corners onto the map is not possible because theeffective tilt angles for at least some of the corners never intersectthe ground.

One illustrative method of resolving this issue, is to determine theupper left 1512 and upper right vertices 1514 of the view cone 1500based upon a resolvable distance of the camera. The resolvable distancecan be calculated using the resolution of the camera. Such an approachis advantageous because it better represents the true view field of thecamera. In one illustrative embodiment, the view cone 1500 is displayedso that objects within the view cone can be identified in the video feed(e.g., objects within the resolvable distance), while objects beyond theview cone cannot be distinguished from the background (e.g., objectsbeyond the resolvable distance). In one illustrative embodiment, theresolvable distance corresponds to a distance at which a 4 meter longobject (approximately the size of a vehicle) corresponds to two pixelson the display. In one embodiment, the resolvable distance is determinedusing Equation 27,

$\begin{matrix}{D = {\frac{\frac{W}{2} \times R}{\tan(\Theta)}}} & (27)\end{matrix}$wherein W is the width of the video feed in pixels, R is the resolutionof the camera in meters per pixel, and Θ is the horizontal angular fieldof view (H_(FOV)) 1520 divided by two. In some embodiments, the width ofthe video feed is communicated to a processor by the camera. In otherillustrative embodiments, the width of the video feed is a fixed number,such as 320 pixels. FIG. 11 shows how the resolvable distance iscalculated using a right triangle in accordance with one embodiment ofthe present invention. The calculated resolvable distance is then usedto determine the upper left 1512 and upper right vertices 1514 of theview cone 1500.

In one illustrative embodiment, the resolvable distance is projectedalong the effective pan angles (θ) for the upper left 1504 and upperright 1506 corners of the video feed. The “map” Cartesian coordinate(X_(P), Y_(P), Z_(P)) for the upper left 1512 and upper right 1514vertices can be determined according to Equations 28, 29 and 30,X _(P)=(sin(θ)×D)+X _(C)  (28)Y _(P)=(cos(θ)×D)+Y _(C)  (29)Z _(P) =Z _(C)  (30)wherein (X_(C), Y_(C), Z_(C)) are the “map” Cartesian coordinates forthe camera. In illustrative embodiments of the present invention, the“map” Cartesian coordinates (X_(P), Y_(P), Z_(P)) for upper left 1512and upper right 1514 vertices are then used to determine and display theview cone on the map. In other illustrative embodiments, these “map”Cartesian coordinates are converted into geospatial coordinates (e.g.,using Equations 22-26) and then used to determine and display the viewcone on the map.

In another illustrative embodiment, instead of projecting the resolvabledistance along the effective pan angles (θ) for the corners 1504 and1506, the resolvable distance is projected along the pan angle (θ) ofthe camera to a resolvable point. In some embodiments, the resolvablepoint is the center of the most distant edge of the view cone 1502. FIG.18 shows a view cone 1800 with the center 1802 of its most distant edgeas the resolvable point, in accordance with one embodiment of thepresent invention. The “map” Cartesian coordinate (X_(R), Y_(R), Z_(R))for the resolvable point can be determined according to Equations 31, 32and 33:X _(R)=(sin(θ)×D)+X _(C)  (31)Y _(R)=(cos(θ)×D)+Y _(C)  (32)Z _(R) =Z _(C)  (33)

The resolvable point (X_(R), Y_(R), Z_(R)) is then rotated left andright by half the horizontal field of view (H_(FOV)) 1520 to determinethe “map” Cartesian coordinates (X_(P), Y_(P), Z_(P)) for upper left1804 and upper right 1806 vertices of the view cone. For the sake ofcomputational efficiency in the processor, a matrix can be used torotate the resolvable point. The “map” Cartesian coordinates (X_(P),Y_(P), Z_(P)) for upper left 1804 and upper right 1806 vertices are thenused to determine and display the view cone on the map. In otherillustrative embodiments, however, the “map” Cartesian coordinates areconverted into geospatial coordinates and then used to determine anddisplay the view cone on the map.

In yet further illustrative embodiments, the projections discussed inEquations 28-33 of this section are performed in the “camera” CartesianCoordinate system and are then converted into the “map” Cartesiancoordinate system using a mathematical transformation (or itstransform).

Illustrative embodiments of the present invention are not limited foruse with cameras that are characterized by pan, zoom, and tiltcoordinates (e.g., pan-zoom-tilt cameras). For example, illustrativeembodiments of the present invention may also be used with fixed camerasor other cameras that have a limited range of motion. Indeed, the abovedescribed methods of calibration and prioritization apply similarly tofixed cameras. Also, according to illustrative embodiments of thepresent invention, the view cone of the fixed camera can be modified asits zoom changes. Various embodiments of the present invention also haveapplicability to other types of cameras, such as cameras that are unableto shift their tilt orientation.

It should be noted that terms such as “system”, “processor”, “server”,“input device”, “display device”, “communications network”, and“database” may be used herein to describe devices that may be used incertain embodiments of the present invention and should not be construedto limit the present invention to any particular device type or systemunless the context otherwise requires. Thus, a system may include,without limitation, a client, server, computer, appliance, or other typeof device. Such devices typically include one or more network interfacesfor communicating over a communication network (e.g., computer network)and a processor (e.g., a microprocessor with memory and otherperipherals and/or application-specific hardware) configured accordinglyto perform device and/or system functions. Communication networksgenerally may include public and/or private networks; may includelocal-area, wide-area, metropolitan-area, storage, and/or other types ofnetworks; and may employ communication technologies including, but in noway limited to, analog technologies, digital technologies, opticaltechnologies, wireless technologies, networking technologies, andinternetworking technologies.

It should also be noted that devices may use communication protocols andmessages (e.g., messages created, transmitted, received, stored, and/orprocessed by the system), and such messages may be conveyed by acommunication network or medium. Unless the context otherwise requires,the present invention should not be construed as being limited to anyparticular communication message type, communication message format, orcommunication protocol. Thus, a communication message generally mayinclude, without limitation, a frame, packet, datagram, user datagram,cell, or other type of communication message. Unless the contextrequires otherwise, references to specific communication protocols areexemplary, and it should be understood that alternative embodiments may,as appropriate, employ variations of such communication protocols (e.g.,modifications or extensions of the protocol that may be made fromtime-to-time) or other protocols either known or developed in thefuture.

It should also be noted that logic flows may be described herein todemonstrate various aspects of the invention, and should not beconstrued to limit the present invention to any particular logic flow orlogic implementation. The described logic may be partitioned intodifferent logic blocks (e.g., programs, modules, interfaces, functions,or subroutines) without changing the overall results or otherwisedeparting from the true scope of the invention. Often times, logicelements may be added, modified, omitted, performed in a differentorder, or implemented using different logic constructs (e.g., logicgates, looping primitives, conditional logic, and other logicconstructs) without changing the overall results or otherwise departingfrom the true scope of the invention.

The present invention may be embodied in many different forms,including, but in no way limited to, computer program logic for use witha processor (e.g., a microprocessor, microcontroller, digital signalprocessor, or general purpose computer), programmable logic for use witha programmable logic device (e.g., a Field Programmable Gate Array(FPGA) or other PLD), discrete components, integrated circuitry (e.g.,an Application Specific Integrated Circuit (ASIC)), or any other meansincluding any combination thereof. In a typical embodiment of thepresent invention, predominantly all of the described logic isimplemented as a set of computer program instructions that is convertedinto a computer executable form, stored as such in a computer readablemedium, and executed by a microprocessor under the control of anoperating system.

Computer program logic implementing all or part of the functionalitypreviously described herein may be embodied in various forms, including,but in no way limited to, a source code form, a computer executableform, and various intermediate forms (e.g., forms generated by anassembler, compiler, linker, or locator). Source code may include aseries of computer program instructions implemented in any of variousprogramming languages (e.g., an object code, an assembly language, or ahigh-level language such as Fortran, C, C++, JAVA, or HTML) for use withvarious operating systems or operating environments. The source code maydefine and use various data structures and communication messages. Thesource code may be in a computer executable form (e.g., via aninterpreter), or the source code may be converted (e.g., via atranslator, assembler, or compiler) into a computer executable form.

The computer program may be fixed in any form (e.g., source code form,computer executable form, or an intermediate form) either permanently ortransitorily in a tangible storage medium, such as a semiconductormemory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-ProgrammableRAM), a magnetic memory device (e.g., a diskette or fixed disk), anoptical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card),or other memory device. The computer program may be fixed in any form ina signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies, networking technologies, and internetworking technologies.The computer program may be distributed in any form as a removablestorage medium with accompanying printed or electronic documentation(e.g., shrink wrapped software), preloaded with a computer system (e.g.,on system ROM or fixed disk), or distributed from a server or electronicbulletin board over the communication system (e.g., the Internet orWorld Wide Web).

Hardware logic (including programmable logic for use with a programmablelogic device) implementing all or part of the functionality previouslydescribed herein may be designed using traditional manual methods, ormay be designed, captured, simulated, or documented electronically usingvarious tools, such as Computer Aided Design (CAD), a hardwaredescription language (e.g., VHDL or AHDL), or a PLD programming language(e.g., PALASM, ABEL, or CUPL).

Programmable logic may be fixed either permanently or transitorily in atleast one tangible storage medium, such as a semiconductor memory device(e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magneticmemory device (e.g., a diskette or fixed disk), an optical memory device(e.g., a CD-ROM), or other memory device. The programmable logic may befixed in a signal that is transmittable to a computer using any ofvarious communication technologies, including, but in no way limited to,analog technologies, digital technologies, optical technologies,wireless technologies (e.g., Bluetooth), networking technologies, andinternetworking technologies. The programmable logic may be distributedas a removable storage medium with accompanying printed or electronicdocumentation (e.g., shrink wrapped software), preloaded with a computersystem (e.g., on system ROM or fixed disk), or distributed from a serveror electronic bulletin board over the communication system (e.g., theInternet or World Wide Web). Of course, some embodiments of theinvention may be implemented as a combination of both software (e.g., acomputer program product) and hardware. Still other embodiments of theinvention are implemented as entirely hardware, or entirely software.

The embodiments of the invention described above are intended to bemerely exemplary; numerous variations and modifications will be apparentto those skilled in the art. All such variations and modifications areintended to be within the scope of the present invention as defined inany appended claims.

What is claimed is:
 1. A computer-implemented method for calibrating atleast one camera, the system including at least one camera, the methodcomprising: displaying a video feed from the at least one camera, the atleast one camera having an orientation characterized by pan, zoom, andtilt coordinates; displaying a map of an area, the map beingcharacterized by geospatial coordinates; allowing a user to select atleast three pairs of points using at least one input device, a firstpoint of the pair being selected in the map and a second point of thepair being selected from the video feed, the first point and the secondpoint corresponding to the same geographic location; converting, in acomputer process, the at least three points selected in the map fromgeospatial coordinates into Cartesian coordinates in three dimensionsdefined by a first three-dimensional coordinate system; converting, in acomputer process, the at least three points selected in the video feedfrom pan, zoom, and tilt coordinates into Cartesian coordinates in threedimensions defined by a second three-dimensional coordinate system; anddetermining, in a computer process, a mathematical rotation between thefirst three-dimensional coordinate system and the secondthree-dimensional coordinate system based upon the Cartesian coordinatesfor the at least three pairs of points.
 2. A method according to claim1, wherein the mathematical rotation is a matrix.
 3. A method accordingto claim 1, further comprising: displaying the location of the at leastone camera on the map.
 4. A method according to claim 1, wherein thegeospatial coordinates are latitude, longitude, and altitudecoordinates.
 5. A method according to claim 1, wherein the input deviceis at least one of a mouse, a cursor, a crosshair, a touch screen, and akeyboard.
 6. A method according to claim 1, further comprising: allowingthe user to select at least one point in the map using the at least oneinput device; converting, in a computer process, the geospatialcoordinates for the selected point into Cartesian coordinates in threedimensions defined by the first three-dimensional coordinate system;applying, in a computer process, the rotation to the Cartesiancoordinates for the selected point to determine Cartesian coordinates inthree dimensions defined by the second three-dimensional coordinatesystem; converting, in a computer process, the Cartesian coordinatesdefined by the second three-dimensional coordinate system into pan andtilt coordinates for the selected point; and providing orientationinstructions to the at least one camera based upon the pan and tiltcoordinates for the selected point.
 7. A method according to claim 1,further comprising: receiving coordinates from a sensor for at least onetarget; if the coordinates for the at least one target are not Cartesiancoordinates defined by the first three-dimensional coordinate system,converting, in a computer process, the coordinates into Cartesiancoordinates in three dimensions defined by the first three-dimensionalcoordinate system; applying, in a computer process, the rotation to theCartesian coordinates defined by the first three-dimensional coordinatesystem to determine Cartesian coordinates in three dimensions defined bythe second three-dimensional coordinate system; converting, in acomputer process, the Cartesian coordinates defined by the secondthree-dimensional coordinate system into pan and tilt coordinates; andproviding orientation instructions to the at least one camera based uponthe pan and tilt coordinates.
 8. A method according to claim 7, furthercomprising: displaying the location of the at least one sensor on themap.
 9. A method according to claim 7, further comprising: displayingthe location of the at least one target on the map.
 10. A methodaccording to claim 1, wherein the video feed has upper left, upperright, lower left, and lower right corners, and the method furthercomprising: determining, in a computer process, effective pan and tiltangles for at least the lower left and lower right corners of the videofeed based upon the pan, zoom, and tilt coordinates for the cameraorientation; converting, in a computer process, the effective pan andtilt angles for at least the lower left and lower right corners of thevideo feed into Cartesian coordinates in three dimensions defined by thesecond three-dimensional coordinate system; applying, in a computerprocess, the rotation to the Cartesian coordinates defined by the secondcoordinate system to determine Cartesian coordinates in three dimensionsdefined by the first coordinate system for at least the lower left andlower right corners of the video feed; determining, in a computerprocess, a view cone using the Cartesian coordinates defined by thefirst coordinate system for at least the lower left and lower rightcorners of the video feed; determining, in a computer process, the viewcone based upon the upper left and upper right corners of the videofeed; and displaying the view cone on the map.
 11. A method according toclaim 10, wherein, when the tilt coordinate for the camera is below thehorizon, determining the view cone based upon the upper left and upperright corners of the video feed comprises: determining, in a computerprocess, effective pan for the upper left and upper right corners of thevideo feed based upon the pan, zoom, and tilt coordinates for the cameraorientation; and converting, in a computer process, the effective panand tilt angles for the upper left and upper right corners of the videofeed into Cartesian coordinates in three dimensions defined by thesecond three-dimensional coordinate system; applying, in a computerprocess, the mathematical rotation to the Cartesian coordinates todetermine Cartesian coordinates in three dimensions defined by the firstthree-dimensional coordinate system for the upper left and upper rightcorners of the video feed; and determining, in a computer process, theview cone based upon the Cartesian coordinates, defined by the firstthree-dimensional coordinate system, for the upper left, upper right,lower left and lower right corners.
 12. A method according to claim 11,wherein the view cone is a polygon and the Cartesian coordinates,defined by the first three-dimensional coordinate system, for the upperleft, upper right, lower left and lower right corners are the verticesof the polygon.
 13. A method according to claim 10, wherein, when thetilt coordinate for the camera is above the horizon, determining theview cone based upon the upper left and upper right corners of the videofeed comprises: determining, in a computer process, effective tiltangles for the upper left and upper right corners of the video feedbased upon the pan, zoom, and tilt coordinates for the camera;determining, in a computer process, coordinates in three dimensions,defined by the first three-dimensional coordinate system, for the upperleft and upper right corners of the video feed based upon a resolvabledistance of the camera; and determining, in a computer process, the viewcone based upon the Cartesian coordinates, defined by the firstthree-dimensional coordinate system, for the upper left, upper right,lower left and lower right corners.
 14. A method according to claim 13,wherein the view cone is a polygon and the Cartesian coordinates,defined by the first three-dimensional coordinate system, for the upperleft, upper right, lower left and lower right corners are the verticesof the polygon.
 15. Apparatus comprising at least one non-transitorycomputer readable medium encoded with instructions which when loaded onat least one computer, establish processes for calibrating at least onecamera orientation characterized by pan, zoom, and tilt coordinates, theprocesses including: allowing a user to select at least three pairs ofpoints using at least one input device, a first point of the pair beingselected in a map characterized by geospatial coordinates and a secondpoint of the pair being selected from a video feed from the at least onecamera, the first point and the second point corresponding to the samegeographic location; converting the at least three points selected inthe map from geospatial coordinates into Cartesian coordinates in threedimensions defined by a first three-dimensional coordinate system;converting the at least three points selected in the video feed frompan, zoom, and tilt coordinates into Cartesian coordinates in threedimensions defined by a second three-dimensional coordinate system; anddetermining a mathematical rotation between the first three-dimensionalcoordinate system and the second three-dimensional coordinate systembased upon the Cartesian coordinates for the at least three pairs ofpoints.
 16. An apparatus according to claim 15, wherein the instructionsestablish processes further including: converting the geospatialcoordinates for a selected point into Cartesian coordinates defined bythe first three-dimensional coordinate system; applying the rotation tothe Cartesian coordinates for the selected point to determine Cartesiancoordinates defined by the second three-dimensional coordinate system;converting the Cartesian coordinates defined by the secondthree-dimensional coordinate system into pan and tilt coordinates forthe selected point; and providing orientation instructions to the atleast one camera based upon the pan and tilt coordinates for theselected point.
 17. An apparatus according to claim 15, wherein theinstructions establish processes further including: receivingcoordinates from a sensor for at least one target; if the coordinatesfor the at least one target are not Cartesian coordinates defined by thefirst three-dimensional coordinate system, converting the coordinatesinto Cartesian coordinates defined by the first three-dimensionalcoordinate system; applying the rotation to the Cartesian coordinatesdefined by the first three-dimensional coordinate system to determineCartesian coordinates defined by the second three-dimensional coordinatesystem; converting the Cartesian coordinates defined by the secondthree-dimensional coordinate system into pan and tilt coordinates; andproviding orientation instructions to the at least one camera based uponthe pan and tilt coordinates.
 18. An apparatus according to claim 15,wherein the instructions establish processes further including:determining effective pan and tilt angles for at least the lower leftand lower right corners of the video feed based upon the pan, zoom, andtilt coordinates for the camera orientation; converting the effectivepan and tilt angles for at least the lower left and lower right cornersof the video feed into Cartesian coordinates defined by the secondthree-dimensional coordinate system; applying the rotation to theCartesian coordinates defined by the second three-dimensional coordinatesystem to determine Cartesian coordinates defined by the firstthree-dimensional coordinate system for at least the lower left andlower right corners of the video feed; determining a view cone using theCartesian coordinates defined by the first three-dimensional coordinatesystem for at least the lower left and lower right corners of the videofeed; determining the view cone based upon the upper left and upperright corners of the video feed; and displaying the view cone on themap.
 19. A surveillance system comprising: at least one camera having anorientation characterized by pan, zoom and tilt coordinates; a processorin communication with the at least one camera; at least one display incommunication with the processor, the at least one display displaying avideo feed from the at least one camera and a map of an area, the mapbeing characterized by geospatial coordinates; at least one input devicein communication with the processor allowing a user to select points onthe video feed and the map; a memory storing instructions executable bythe processor to perform processes that include: converting pointsselected in the map from the geospatial coordinates into Cartesiancoordinates in three dimensions defined by a first three-dimensionalcoordinate system; converting points selected in the video feed frompan, zoom, and tilt coordinates into Cartesian coordinates in threedimensions defined by a second three-dimensional coordinate system; anddetermining a mathematical rotation between the first three-dimensionalcoordinate system and the second three-dimensional coordinate systembased upon the Cartesian coordinates for at least three pairs of points,a first point of the pair having been selected in the map and a secondpoint of the pair having been selected from the video feed.
 20. Thesurveillance system of claim 19, wherein the mathematical rotation is amatrix.
 21. The surveillance system of claim 19, wherein the geospatialcoordinates are latitude, longitude, and altitude coordinates.
 22. Thesurveillance system of claim 19, wherein the at least one input deviceis at least one of a mouse, a cursor, a crosshair, a touch screen, and akeyboard.
 23. The surveillance system of claim 19, wherein the memoryfurther stores instructions executable by the processor to performprocesses that include: applying the mathematical rotation to theCartesian coordinates defined by the first three-dimensional coordinatesystem for a point selected by the user in the map to determineCartesian coordinates defined by the second three-dimensional coordinatesystem; converting the Cartesian coordinates defined by the secondthree-dimensional coordinate system into pan and tilt coordinates forthe selected point; and providing orientation instructions to the atleast one camera based upon the pan and tilt coordinates for theselected point.
 24. The surveillance system of claim 19 furthercomprising a sensor connected to provide coordinates to the processor.